fix(eks): set resource requests/limits for Calico/Tigera components - #343
Conversation
The tigera-operator chart ships resources: {} and the operator's
built-in component defaults set no memory bounds, so the Calico pods
had no guaranteed reservations and no ceilings — a source of resource
contention on busy nodes.
Pin predictable reservations on the operator pod, calico-node,
calico-typha, calico-kube-controllers, and calico-apiserver, following
a memory-bounded / CPU-unbounded policy: memory request == limit
(non-compressible; guaranteed floor + bounded ceiling), CPU request
only with no CPU limit (avoids CFS-throttling the dataplane).
EKS only; AKS uses the managed Calico add-on.
|
@claude review |
|
Claude finished @amdove's task in 1m 0s —— View job Code Review
This is a clean, well-scoped change. The implementation is correct and the rationale is sound. A few observations:
The two helpers are a nice abstraction. One minor note: The Resource values The chosen values look reasonable for typical cluster sizes. One thing to verify if you haven't already: the tigera-operator chart's
Overall No issues blocking merge. The change is correct, well-tested, and appropriately scoped to EKS. The AKS limitation is clearly documented. Approved by stevenolen — LGTM. |
Address review feedback: note that calicoComponentOverride patches only the main container (not ephemeral initContainers like install-cni), and explain why calico-apiserver's resources live under apiServer rather than installation (they map to the separate APIServer CR).
…nale The tigera operator ships no resource defaults of its own — an Installation without the deprecated ComponentResources renders empty resources — so the previous comment claiming calico-node "keeps the operator's built-in 250m CPU request floor" was wrong. 250m is the request upstream sets on calico-node in the self-managed manifests/calico.yaml; under the operator it is a new per-node reservation. Comment corrected to say so. Also bounds csi-node-driver, which was still running unbounded on every node: the operator enables the Calico CSI plugin unless kubeletVolumePluginPath is "None", and PTD leaves it at the default. Both of its containers (calico-csi and csi-node-driver-registrar) are small and idle once the driver registers with kubelet, so they take the smallest bounds of any component here. calicoComponentOverride now takes a variadic container list (via a new calicoContainer helper) to express the two-container CSI DaemonSet, and the test looks containers up by name instead of by position.
|
@claude review again please |
|
Claude finished @amdove's task in 54s —— View job Code Review
The implementation is correct and well-structured. No issues blocking merge. Here are the findings:
The three helpers (
The Resource values The values are defensible.
Overall No blocking issues. Implementation matches the schema, policy is correctly encoded and tested, and the scoping rationale (EKS fixed, AKS out of scope) is clear. |
Measured peak working set across the fleet over 7 days showed two components sitting at ~1.04x their ceiling, close enough to OOMKill on any growth: tigera-operator 245.5 MiB against 256Mi calico-kube-controllers 123.5 MiB against 128Mi Both are raised to ~1.5x observed peak (384Mi and 192Mi). Both are single-replica Deployments, so neither adds anything to the per-node reservation, which stays at 512Mi for calico-node. calico-node keeps 512Mi: at 374.8 MiB observed that is already ~1.37x, and raising it would reserve another 256Mi on every node. typha (1.94x) and apiserver (3.34x) already clear the bar and are unchanged. tigera-operator's CPU request goes 100m -> 250m; it peaks near 350m and is a single pod, so the reservation is negligible. calico-node stays at 250m despite peaking near 750m: with no CPU limit that peak isn't throttled, and raising the request would cost capacity on every node. ~1.5x rather than 2x because request == limit makes headroom reserved capacity rather than a free kill threshold. Also corrects the csi-node-driver comment: it does not run on every node today. The operator enables CSI when kubeletVolumePluginPath is unset (which PTD leaves it as), but existing clusters carry an out-of-band "None" that disables it, so the override is inert until that is resolved.
The comment claimed all bounds were ~1.5x observed peak, but only the two that were raised land there. 1.5x was the threshold for deciding what needed headroom, not a target every value hits: typha and apiserver were already above it and left alone, and calico-node sits at ~1.37x deliberately because it is the only per-node DaemonSet.
The multipliers aren't a design target and the cost of memory doesn't vary by component — each bound is just a round value above the measured peak. Says that instead.
2 participants
Description
Sets resource requests and limits on the Calico/Tigera components installed by the
tigera-operatorHelm chart on EKS. Previously the chart shippedresources: {}and the operator has no defaults of its own, so these pods had no reservations and no ceilings and competed unbounded with workloads on busy nodes.Policy:
calico-csi,csi-node-driver-registrar)Nothing was reserved before this, so these are new reservations rather than adjustments. Reserved memory added per node: 512Mi —
calico-nodeis the only per-node DaemonSet running, and everything else is a single-replica Deployment that lands on one node.Sizing
Memory bounds are round values above the highest working set observed across the fleet over 7 days (control-room Mimir,
max_over_time(container_memory_working_set_bytes[7d])across the Calico namespaces):tigera-operatorandcalico-kube-controllerswere originally 256Mi and 128Mi, within 4% of their observed peaks, and were raised once measured. The others already had room and are unchanged.Headroom is deliberately modest rather than a round 2×, because request == limit makes it reserved capacity, not just a kill threshold.
calico-nodehas the least (~1.4×) and is the one to watch, since it's the only per-node DaemonSet and any increase lands on every node.CPU requests are floors, not caps, since nothing sets a CPU limit.
calico-nodepeaks near 750m against its 250m request and stays there — raising it would reserve capacity on every node without preventing anything.tigera-operatorpeaks near 350m and goes to 250m.The 250m on
calico-nodematches upstream: across the self-managed install manifests at v3.31.4 it's the only resource value set on any component, with no memory value anywhere. projectcalico#5418, asking for recommendations, was closed as not planned.Code Flow
All in
deployTigeraOperator(lib/steps/eks_helpers.go).calicoResourcesbuilds the{requests: {cpu, memory}, limits: {memory}}block;calicoContainerpairs a container name with it;calicoComponentOverridewraps those in the operator's strategic-merge shape (variadic, sincecsi-node-driverhas two containers).Values map to
installation.calicoNodeDaemonSet/typhaDeployment/calicoKubeControllersDeployment/csiNodeDriverDaemonSet, plus top-levelresourcesfor the operator pod andapiServer.apiServerDeploymentfor calico-apiserver.csi-node-driver is bounded but currently inert. The operator enables Calico CSI whenever
installation.kubeletVolumePluginPathis unset, which is what PTD leaves it as, so a cluster built today runs this DaemonSet on every node. Existing clusters carry an out-of-band"None"that disables it, and nocalico-csicontainer appears in fleet metrics. Ownership of that field is tracked separately.Scope
EKS: fixed. PTD installs the chart here, so the reservations attach directly to the release.
AKS: not fixed. AKS Calico comes from the Azure-managed network-policy add-on. Anything in
kube-systemis Microsoft-managed and customers can't alter it, so the Installation CR is Azure's to reconcile. Fixing it there means migrating to self-managed Calico or moving the dataplane to Cilium, which Microsoft recommends — otherwise node right-sizing is the only lever.Watch after applying
calico-nodehas the thinnest margin and its footprint grows with endpoints/policies.Category of change
Checklist
Testing
just format,go vet ./..., andjust test-libpass on the branch merged with main.TestAWSEKSDeployTigeraResourceRequestsasserts each component's values and the no-CPU-limit invariant.